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ABSTRACT 

Peer review is by no means a routine process for traditional, or basic, research. Even so, 
peer review is even less routinized for other forms of scholarship. (In 1990, Ernest Boyer called 
for a reconsideration of scholarship and extended the definition to be inclusive of non-traditional 
modes of scholarly production and delivery. However, peer review processes for non-traditional 
scholarship modes have proven difficult to assess and implement. An examination of promotion 
and tenure documents at a regional comprehensive university reveals the various strategies 
departments use to provide peer review for work faculty consider to be non-traditional. The 
study found five models for peer review of non-traditional scholarship that have implications for 
other institutions seeking to recognize and reward non-traditional scholarship. 
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INTRODUCTION 

Kerry Ann O’Meara and Eugene Rice interviewed Chief Academic Officers at over 700 
institutions regarding teaching, research and service throughout the 1990s (O’Meara, 2005). 

Their findings show that the majority report that publication productivity counted more at the 
end of the decade than at the beginning. Coincidentally, research associated with engagement, 
teaching, and professional work was introduced by Boyer (1990) as valid forms of scholarship 
that should be rewarded at academic institutions. Prior to rewarding faculty members for their 
broadened scholarly work, institutions must first participate in the assessment of that work. The 
most common and most accepted form of assessment for scholarly work involves the double¬ 
blind editorial peer review process. How does the peer review process for the non-traditional 
scholarship play out for faculty? How has the process been institutionalized in documents such 
as the tenure and promotion documents that serve as agreements between faculty members and 
their departments? 

Peer review 

The scholarly academies of the eighteenth century pioneered the use of peer review in 
evaluating each other’s work (Drummond, 1999). At first confined to members, the use of 
outside reviewers became common practice by the late nineteenth century, largely due to the 
growing volume of scholarship being produced and disseminated (Spier, 2002). Single-blind and 
double-blind systems intermingled until after World War II, when the double-blind process, in 
which the authors do not know their reviewers and the reviewers do not the authors, prevailed. 
Proponents of the system claim that it provides the most reliable, valid and objective means by 
which academic scholarship can be assessed (Bedeian, 2004; Ware, 2008; Weller, 2001). 

Indeed, the prepublication, double-blind peer review system has evolved into the standard 
for academic work today and a prominent journal editor has labeled it “absolutely sacred” 
(McCook, 2006). The system, though, is not without its critics (Suls and Martin, 2009). Since the 
1980s, the system has received a great deal of criticism for being a non-standardized, unreliable 
and unfair way to validate the quality of the manuscript and a process which often benefits 
prominent researchers (Horrobin, 1982: Jefferson, 2002; Lock, 1985; Shatz, 2004; Smit, 2006; 
Starbuck, 2003; Weller, 2001; Wenneras & Wold, 2007). Further, from the production side, the 
process has been labeled unwieldy, expensive, and ineffective (Rennie, 1999; Okerson & 
McDonald, 1995). Finally, peer review has been shown to be inherently conservative, favoring 
conventional practice over innovation (Epstein, 1995; Forsdyke, 2007; Mahoney, 1977). 

Because the process has become so closely intertwined with definitions of quality, efforts 
to revise or expand peer review mechanisms have met with considerable resistance from both 
within and outside of academia. Faculty, according to Eugene Rice, tend to live in an 
“assumptive world” and the identification of quality with the double-blind peer review process is 
one of the primary assumptions upon which the review of scholarship is based (Rice, 1996). In 
other words, without peer review at its base, the entire edifice of scholarly publication, even 
academic culture, finds itself on highly unstable ground. The impetus for non-traditional forms 
of scholarship only further destabilizes the ground on which peer review rests. 
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Non-traditional scholarship and peer review 

For most disciplines, traditional scholarship involves print publication via established 
journals or texts produced by reputable publishers. These venues have long been the bastions of 
the double-blind peer review process and represent lengthy and recognized traditions for 
establishing quality control. While there are some disciplinary exceptions (such as performance 
in the arts), the double-blind process remains the gold standard by which the quality of academic 
work is measured. One of the proposed benefits of Ernest Boyer’s model of scholarship is that it 
opened the doors to a wider range of faculty work being recognized and rewarded as scholarship. 
That range does not simply include the recognition of different types of scholarly activities, but 
also different scholarly products and/or different means of scholarly communication, many of 
which the traditional peer review system is ill-equipped to handle (Reuter & Bauer, 2005). 

The pioneers of the Boyer scholarship model emphasized the need for largely 
conventional peer review processes as the mechanism through which the newly-recognized 
forms would be assessed and validated (Fincher & Work, 2006; Shulman, 1999). This has 
proven to be easier to do in theory than in practice (Schweitzer, 2000) and scholars have debated 
whether the problem is with the Boyer model, the peer review process, or both (Boshier, 2009). 

After Boyer identified the four types of scholarship, other colleagues at the Carnegie Institute 
began the work of studying how they could be assessed. They proposed a review model that 
defined high-quality scholarly work as possessing clear goals, adequate preparation, appropriate 
methods, outstanding results, effective communication, and a reflective critique (Glassick, Huber 
& Maeroff, 1997; Glassick, 2000). While this model addresses the need for clear standards of 
review, the basis of it was a distillation of the standards used by traditional peer-review outlets, 
including major publishers, journal editors, and grant agencies. On the subject of the special 
issues inherent in non-traditional scholarship, the authors certainly acknowledge the issue, 
commenting that “it takes imagination” to document non-traditional scholarship (Glassick et al, 
1997). That being said, their solutions largely focus on creating portfolios to capture the 
scholarship of teaching and learning and they do little more to discuss non-traditional products 
that stem from other areas of Boyer scholarship, such as application/engagement or integration. 
This study shows how one university attempted to reconcile the standards set by Scholarship 
Assessed , the Boyer model of scholarship, non-traditional scholarly products, and peer review 
across all disciplines. By examining the tenure and promotion documents at a regional 
comprehensive university, the researchers reveal the various strategies of peer review that 
worked for non-traditional scholarly products. 

RESEARCH METHODOLOGY 

Research Sample 

Those who have advocated for an expansion of peer-review processes to include non- 
traditional products have strongly suggested that such cultural changes must first begin at the 
campus level and must take into account not only a variety of disciplinary cultures, but also 
institutional cultures (Bergquist, 1992; Diamond, 1999; Kezar & Eckel, 2002; Schein, 1992). For 
the purposes of this study, the institutional culture under consideration is Western Carolina 
University. Begun in the nineteenth century as a teachers’ college, Western Carolina University 
has evolved into a medium-sized, regional comprehensive university, ranked Masters Level II in 
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the Carnegie classification system. Not dissimilar to land grant universities, regional 
comprehensive institutions often have explicit community engagement missions which can make 
the adoption of broader definitions of scholarship more attractive (Finnegan & Gamson, 1996; 
O’Meara & Jaeger, 2007). Located in a rural area, Western Carolina University is no exception 
and has recently received the “engaged” designation from Carnegie in recognition of its mission 
in regional development. 

In 2008, Western Carolina University formally adopted the Boyer model of scholarship 
into its recognition and reward systems. Using a template provided by the Office of the Provost, 
departments were empowered to translate the Boyer directive through the lens of their respective 
disciplinary cultures. The Provost, in collaboration with the Faculty Senate, affirmed the 
principle of departmental autonomy, requiring the new documents only to recognize the four 
types of Boyer scholarship in some way and to provide some kind of mechanism for peer review 
of non-traditional scholarship. Curious about the results of the process, a small group of 
researchers from the Western Carolina University faculty examined the ways peer review for 
non-traditional scholarship is implemented in departments across campus through these revised 
tenure and promotion documents (abbreviated CRD for Collegial Review Document). For the 
sake of the study, this includes 33 of these documents (all departments on campus) effective in 
the Fall of 2008. 

Coding 

The template requires evaluation of three typical domains: teaching, scholarship and 
creative works, and service. Each domain requires that the department specify the methods used 
for evaluation and sources of evidence that are appropriate for each domain. Generally, each 
department’s expectation for peer review for the scholarship of application appeared in the 
methods and sources section of the Scholarship and Creative Works domain. 

RESULTS 

Prior Review Procedures for Scholarship of Engagement Scholarly Activities 

Because these processes, indeed these scholarly products, are new, approximately one- 
third of all departments at Western Carolina University (10 out of 33) either required or strongly 
suggested that faculty discuss non-traditional projects with either the department head, dean, or 
collegial review committee prior to undertaking the scholarly activity. Prior review was most 
prevalent in the Western Carolina University School of Engineering and Technology (2 of 2 
departments) and in the College of Arts and Sciences (3 of 11 departments). These peer review 
processes are varied and involve internal and external reviewers and range from quite informal to 
more formal processes. 

When the peer review processes for internal review involved internal reviewers, only 
rarely was that internal reviewer a department head. However, one CRD document suggested 
that faculty members consult with their department head for feedback as to the Boyer 
classification of the proposed scholarly activity: 

"Discipline-specific proprietary work may [be considered scholarship of application or 
engagement] in which case, appropriate discussion should be had with the department head, 
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early in the process to determine how the work will be classified." (pg. 3, Department of 
Accounting, Finance, Information Systems and Economics, College of Business) 

Beyond simple classification of the scholarly activity, a typically an informal 
conversation with a department head would likely also include other project aspects as well. 
While processes involving feedback from department heads were only mentioned once, 
processes with feedback from groups or committees were much more frequent. Generally, when 
processes involved only internal reviewers providing pre-project feedback, the reviewers were 
identified as those serving on the departmental collegial review committee. Representative 
statements from the CRD documents appear below: 

“The candidate may request prior review of the proposed project in order to get feedback 
from the Collegial Review Advisory Committee” (pg. 15, Department of Early and Middle 
Grades Education, College of Education and Allied Professions). 

“Faculty who seek to pursue projects that will require customized peer review should make 
the head of the department aware at an early stage and should receive regular feedback from 
the department personnel committee to ensure that both the individual’s and the 
department’s expectations for the scope, the viability, and the scholarly quality of the project 
are maintained” (pg. 9, Department of English, College of Arts and Sciences). 

In 40% of the cases (4 out of 10), pre-project feedback procedures explicitly mention 
external reviewers. 

“The candidate may request a prior review of the proposed project in order to get feedback 
from the TPR Advisory Committee. The TPR Advisory Committee may solicit outside 
reviewers if necessary. In these cases of prior review, the TPR Advisory Committee will 
provide written feedback to the faculty member for inclusion in their dossier or other 
evaluation materials, and a copy will also be provided to the Department Head for placement 
in the faculty member’s departmental file” (pg. 7, Department of Communication, College 
of Arts and Sciences). 

“In cases where the candidate seeks prior review of a proposed project, the CRD committee 
should consult with experts at peer institutions to perform the peer review function, but 
would supply candidate with written feedback, so as to be included in dossier or other 
evaluation materials and a copy would be submitted to dept head for candidate's files” (pg. 

6, Department of Philosophy and Religion, College of Arts and Sciences). 

One school, the Western Carolina University School of Engineering and Technology, 
created a formal Engagement Committee, charged with reviewing projects involving the 
scholarship of engagement. The Engagement Committee, which includes both internal and 
external members, is appointed by the Dean and serves at the discretion of the Dean. A minimum 
of two of the Engagement Committee members also serve on the college Collegial Review 
Committee. Both departments in the Western Carolina University School, Construction 
Management and the Department of Engineering and Technology, use identical language in their 


Bound by tradition?, page 5 


Research in Higher Education Journal 


CRD documents to outline procedures for prior review of scholarly activities by the Engagement 
Committee: 

“Specifically, the [Engagement] Committee is responsible for ... pre-screening of 
engagement activities to ensure that they match the mission of the department and the 
resources available [and] assistance in developing roadmaps for success in the planning, 
implementation and documentation of Scholarship of Application programs...” (pg. 26, 
Department of Engineering and Technology, Kimmel School of Engineering and 
Technology). 

While the documents do not specifically state the reasons for encouraging or requiring prior 
review, it can be surmised that the process is designed to address the risks inherent in pursuing 
new forms of scholarship by establishing clear expectations, on both sides, of where and how the 
scholarly activity will be recognized and rewarded. This practice is also in keeping with the 
recommendations made by Kerry Ann O’Meara and Eugene Rice as a result of their national 
study on redefining scholarship (O’Meara and Rice, 2005). 

Review Procedures 

Internal Review 

The Office of the Provost initially specified only that the CRD documents contain a 
mechanism for review of non-traditional scholarship. In a handful of cases (2), departments 
determined that internal review would be sufficient: 

"Evidence of scholarship may be evaluated by the Department Head, and if applicable, by a 
Peer Evaluation Committee of tenured faculty members within the Department of the faculty 
member being evaluated” (pg. 3. Department of Modem Foreign Languages, College of Arts 
and Sciences) 

“The Committee is organized and managed at the direction of the Department Head. The 
Committee has both pre-screening and post-review evaluation responsibilities over 
Scholarship of Application programs. Specifically, the committee is responsible for: Pre¬ 
screening of engagement activities to ensure that they match the mission of the department 
and the resources available ... Validation as Scholarship of Application Additionally, the 
results of scholarly engagement may be documented through report(s) from stakeholders 
and/or clients documenting the magnitude of delivery and/or impact resulting from the 
engagement activities and application of disciplinary expertise” (Appendix D, School of 
Physical Therapy, College of Health and Human Services). 

Upon review of all of the CRD documents in their entirety, the Provost determined that some 
form of external review would be required and asked that these departments to revise their 
standards accordingly. 
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External Review 

Initially, most departments (31 out of 33 or 94%) required external evaluation for peer 
review. The researchers discerned that the departments at Western Carolina University created 
five processes, or models, for peer review of non-traditional scholarship, detailed below. 


Figure 1: The Five Processes of Peer Review for the Scholarship of Engagement and 
Departments within Colleges Following Each Process 
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Library 


LIB-Library 





5 (15.2%) 

9(27.3%) 

8 (24.2%) 

5 (15.2%) 

4(12.1%) 


Model 1: Faculty Documentation of Scholarly Activities 

Several departments (5 out of 33 or 15%) chose to allow faculty candidates to determine 
their own processes for review and to determine an appropriate defense for the quality of their 
scholarship. Typical statements from departmental CRD documents appear below. 

“It is recognized that infrequently a candidate may present “interesting things” that do not fit 
well with these categories yet are still legitimate scholarship. It will be up to the candidate to 
defend the activities as scholarship based on their extraordinary nature.” (pg. 9, Department 
of Business Administration and Law and Sport Management, College of Business) 

“It will be up to the candidate to defend the activities as scholarship based on their 
extraordinary nature and process of external peer review, or justifying why an activity 
should be moved to a higher classification.” (pg. 4, Department of Criminology and 
Criminal Justice, College of Health and Human Services) 

“In cases where scholarly work does not fit the traditional academic peer review model it is 
the responsibility of the faculty member to document in writing how their scholarship fits 
into the model.” (pg. 4, Department of Biology, College of Arts and Sciences) 

“Faculty should document whether their scholarly activities are peer reviewed, invited, peer 
evaluated, or anonymous peer review or evaluation. Faculty should also note if publications 
were subject to an editorial board or other editorial review.” (pg. 6, School of Nursing, 
College of Health and Human Services) 

“When there is no traditional peer review process, the faculty member must document how 
the work will advance the discipline, how the work has been reviewed, and how it will be 
disseminated.” (pg. 3, Department of Geology and Natural Resources, College of Arts and 
Sciences) 

“External review of activities is required for consideration of reappointment, tenure, 
promotion, post tenure review and merit pay. ... Faculty members are required to list 
scholarly activities and note those activities that meet the criteria for scholarship through 
external peer review.” 

In sum, these departments chose to handle non-traditional scholarship on a case-by-case 
basis, giving the burden of proof of external review to the faculty member wishing to pursue 
such scholarship. 
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Model 2: Oven Process 

In 27% of the cases (9 out of 33), departments specified external reviewers but did not 
indicate a specific process for determining how the external reviewers would be determined. 
Representative statements of this process include the following. 

“.. .scholarship must be externally reviewed by a recognized organization.” (pg. 3, 
Department of Math and Computer Science, College of Arts and Sciences) 

“Peer review can include traditional forms (e.g., journal reviewers, editors, committees 
awarding grants), but it can also include a broader community of scholars.” (pg. 3, 
Department of Chemistry and Physics, College of Arts and Sciences) 

“All scholarship must be peer reviewed. We define peer review as the evaluation of the 
scholarly work by people external to Western Carolina University with knowledge and 
expertise in the discipline in order to determine the quality of the work; and where the work 
is made known to the faculty member and others, as appropriate for the work being 
evaluated.” (pg. 4, Department of Biology, College of Arts and Sciences) 

“Peer reviewed works for all forms of scholarship will include those reviewed by one or 
more qualified professionals, external to WCU, in the applicant's field of expertise.” (pg. 3, 
Department of Human Services, College of Education and Allied Professions) 

Unpublished scholarly activities meet the definition of scholarship if they appear in a 
publicly observable form; in other words, it must be public, subject to critical review, and in 
a form allowing the use and exchange by other members of the discipline (Shuler & 
Hutchings, 1998). Unpublished scholarly activity can take the form of a paper, poster, an 
audio or videotape presentation, written report, or Web site (Braxton & Del Favero, 2002). 
(pg2, School of Health Sciences , College of Health and Human Services)( 

In all cases, externally peer reviewed outcomes are most highly valued in all four 
scholarship areas, (pg. 3, School of Health Sciences, College of Health and Human Services) 

All scholarly activities of library faculty that result in the production of scholarship, 
regardless of Boyer category (Boyer, Ernest L. Scholarship Reconsidered: Priorities of the 
Professoriate. Princeton, N.J.: Carnegie Foundation for the Advancement of Teaching, 
1990), are assessed against the following general criteria: (1) the scholarship is subjected to 
peer review. External peer review may be provided by a number of sources outside of the 
Western Carolina University community (pg. 5, ZZZ Library) 

Scholarship that has no obvious external peer review structure in place undergoes peer 
review by a separate process. For such scholarship, the library identifies qualified library 
professionals outside ZZZZ Library with recognized professional standing in the relevant 
area of scholarly activity and requests independent reviews of the quality and impact of the 
scholarship in question, (pg. 6, ZZZ Library) 

Several departments in the College of Business had nearly identical wording: 
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"To count within the category, the activity must lead to an artifact that is evaluated by 
discipline experts who agree the work is a quality expression of one of Boyer’s forms of 
scholarship...(pg. 4, Department of Professional Sales & Marketing and Hospitality and 
Tourism, College of Business) 

"To count within the category, the activity must lead to an artifact that is evaluated by 
discipline experts, external to the University, who agree the work is a quality expression of 
one of Boyer’s forms of scholarship....” (pg. 3, Center for Entrepreneurship and Innovation, 
College of Business) 

These same two departments, Center for Entrepreneurship and Professional Sales, Marketing and 
Hospitality and Tourism, had identical wording regarding the possible makeup of the reviewers. 

“External critical review must be provided by either an accepted review process at a 
recognized journal in the discipline or related discipline, by a sponsoring agency providing 
funding or oversight, the university administration or university system administration or 
other recognized entity outside the department with the expertise to critically evaluate the 
artifact without bias.” (pg. 4, Professional Sales, Marketing and Hospitality and Tourism, 
College of Business) 

For many disciplines, non-traditional scholarship is new territory and without precedents 
to draw from, departments often flexibility over specificity. Those departments in this category 
chose to emphasize the principle of external peer review, but left the practice, or process, of 
selection and evaluation open. This emphasis on external peer review separates these 
departments from those in the previous category, but others chose to provide more explicit 
guidelines for deciding who chooses reviewers and how they are to be selected. 

Model 3: Reviewer Selection with Faculty Involvement 

Many departments (8 out of 33, or 24%) chose to establish external peer review processes 
that included faculty involvement. Typically, that faculty involvement meant that the faculty 
member selected three to five potential reviewers of which the department head, often acting in 
concert with the departmental Collegial Review Committee, selected at least one to serve as an 
external reviewer along with another reviewer selected without faculty involvement. Thus, the 
faculty member generally has the power to select half the reviewers who will formally review the 
non-traditional scholarly project. 

The majority of departments in the College of Education and Applied Professions (3 out 
of 5, or 60%); including Education Leadership Foundations, Health and Physical Education, and 
Psychology presented nearly identical statements regarding faculty involvement in the external 
reviewer selection process: 

"at the time of the third-year reappointment...the candidate will submit to the department 
head up to five names and contact info for potential external reviewers. All of the potential 
reviewers should have expertise in the candidate's discipline, hold the terminal degree in the 
field, and be employed (or have been employed) as a faculty member in an institution of 
higher education. Familiarity with Boyer's model of scholarship is expected. The department 
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head will identify two external reviewers who agree to review the candidate’s materials. At 
least one of those reviewers should be from the list submitted by the candidate. If the other 
reviewer is not on the candidate's list, s/he should have the same qualifications as described 
[above]" (Appendix E, Department of Education, Leadership and Foundations, College of 
Education and Applied Professions). 

In one of the eight cases in this group, faculty members have the ability to select more 
than half of the team of reviewers who will ascertain the quality of their scholarship of 
engagement projects and products. 

“The standard for customized types of peer review will be a panel of 3 external reviewers, 
who will write a substantive peer review of the project that will become part of the CRD 
dossier. Faculty will submit a list of five potential reviewers to the Personnel Committee; if 
the faculty member so desires, two other potential reviewers may be specifically excluded 
from the panel... .The Head and the Committee have the right but are not required to select a 
third reviewer not on the list of potential reviewers” (pg. 9, Department of English, College 
of Arts and Sciences). 

In two departments, the candidate selects all the reviewers who will review the work. 

“External reviewers will be approved by the Department Head after selection by the 
candidate.”(pg. 7, Department of Communication Sciences and Disorders, College of 
Health and Human Sciences) 

“For such activities, taken individually or as a whole, appropriate peer review may be 
established by letters of support from peers external to the university who have personal 
knowledge of specific demonstrations of a candidate’s expertise and who document this 
knowledge in their letters (i.e., the letter must be more specific than a general letter of 
recommendation and must clearly state the nature of the demonstration of expertise to 
which the writer was witness, along with the date and venue of the presentation). External 
peer review may also be established by inviting, at the discretion of the Candidate, 
Director, Dean or the Collegial Review Committee with regard to how that candidate’s 
record would compare to candidates for a similar personnel action at the peer institution. 
If an external reviewer is invited, the Candidate may submit the names of up to three peer 
institutions and up to three individuals whom he or she considers to be peers, from which 
the Collegial Review Committee in consultation with the Director and Dean shall select 
the external reviewer. The external reviewer may, at the discretion of the Director, Dean 
and the Collegial Review Committee, participate in the actual deliberations of the 
Committee, but without a vote. It is up to the candidate to make the case that his or her 
scholarship and creative activity has been appropriately reviewed by peers external to the 
University.” (pg. 6, School of Music, College of Fine and Performing Arts) 

In this model, the process grants the faculty member some agency. While more limited 
than in previous models in which faculty documented the external review, this process allows the 
faculty member to assist the members of the Collegial Review committee in identifying qualified 
reviewers because the level of specialization, even within departments, can make such 
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identifications challenging. Therefore, it is not surprising to see especially diverse departments 
appear in this category. 

This process may also look familiar to those outside of Western Carolina University, as 
this model is not dissimilar to those used by other institutions when considering an applicant for 
tenure. While Western Carolina University does not require external review of a scholarly body 
of work for tenure, these statements indicate that the faculty at Western Carolina University are 
familiar with these procedures and are co-opting them for the purposes of evaluating non- 
traditional scholarship. These procedures are not without controversies of their own, many of 
which resonate with critiques of peer review more broadly (Rhodes-Catanach & Stout, 2002; 
Schlozman, 1998; Schwartz & Schroeder, 1997) but it does allow departments to use a 
recognized and established procedure, albeit with a slightly different objective. 

Model 4: Reviewer Selection by Committee 

Other departments at Western Carolina University (5 out of 33, 15%) chose to specify 
peer review processes that did not include faculty involvement. Rather, these processes left the 
selection of external reviewers to the Collegial Review committees, usually in consultation with 
the department head. 

“The department’s TPR Advisory Committee will judge whether a unit has been achieved 
on a case-by-case basis...”(pg. 4) “Using these general guidelines, the department’s TPR 
Advisory Committee will determine “unit” totals for each faculty member being reviewed, 
(pg. 5) “The TPR Advisory Committee may solicit outside reviews if necessary.” (pg. 7, 
Department of Communication, College of Arts and Sciences) 

“For scholarly work with no extant peer review process, the Department Head, in 
consultation with the Department Collegial Review Committee, will create a formal peer 
review process which will result in at least two written assessments of the work in question 
by qualified external reviewers. These assessments will be filed in the Department office 
and will be included in tenure and promotion dossiers.” (pg. 4, Department of History, 
College of Arts and Sciences) 

“Peer review will include traditional forms (joumal/book reviews, editors) but can also 
include broader communities of scholars (retired professionals, invited addresses at 
conferences or other academic or professional institutions) as recognized by the 
department.. .(pg. 4). Were a candidate to explore scholarly activities that would fit in some 
of the other Boyer categories, the department would count them, insofar as the candidate 
could defend and document those activities as genuine scholarship. Should it be needed, the 
CRD Committee will consult outside reviewers for advice.” (pg. 6, Department of 
Philosophy and Religion, College of Arts and Sciences) 

The department of Anthropology and Sociology and the department of Political Science 
and Public Affairs had identical wording 

“In the case of scholarship where a traditional external review is not possible, the 
Department Head in consultation with the Departmental CRC will create a formal peer 
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review process which will result in at least two positive written assessments of the work in 
question by qualified external reviewers. These assessments will be filed in the departmental 
office and will be included in tenure and promotion dossiers.” (pg. 5, Department of 
Political Science and Public Affairs, College of Arts and Sciences) (pg. 3-4, Department of 
Anthropology and Sociology, College of Arts and Sciences) 

The double-blind peer review process enjoys its hallowed status due, in some part, to its 
emphasis on objectivity. In this process, objectivity is privileged and maintained in two ways. 
First, the faculty member does not participate in the selection of external reviewers, thus 
removing potential bias, and the committee and department head must work together to 
determine reviewers, thus controlling for potential bias on the part of individual department 
members. The latter check and balance is removed in the next process. 

Model 5: Reviewer Selection by Department Head or Dean 

Four departments (4out of 33, or 21%) specified an external peer review process that did 
not include either the faculty member or the department head. In these cases, the selection of 
external peer reviewers was left to the discretion of the department head and/or dean of the 
college. Typical statements from these departments appear below: 

“When scholarship is non-traditional (e.g. scholarship other than traditional paper 
presentations and journal articles), at least two reviewers external to the university will be 
selected by the department head.” (pg. 2-3, Global Strategy and Management, College of 
Business) 

“Members of the Western Carolina University School Engagement Committee serve at the 
discretion of the dean. This committee may have and normally will have both internal and 
external members. The dean serves as the chair.... A minimum of two members of the 
Western Carolina University School tenure and promotion committee also serve on the 
committee.... Due to the proprietary nature of some engagement projects, peer review and 
dissemination may be restricted to the committee and the client's organization.” (pg. 26, 
Department of Engineering and Technology, Western Carolina University School of 
Engineering and Technology) 

“.. .To facilitate the development and evaluation of these [Scholarship of Application] 
activities, the Department Head convenes the ...Collegial Review Committee as required. 
The committee is organized and managed at the direction of the Department Head.... The 
committee is responsible for pre-screening of engagement activities....” 

(pg. 35, School of Physical Therapy) 

O’Meara notes that faculty from master’s level institutions tend to be more wary of 
administrative directives and control (O’Meara, 2006), so it is somewhat surprising to find this 
model in place at Western Carolina University. That being said, it should be noted that both 
departments in the Western Carolina University School of Engineering chose this process, 
perhaps a reflection of the relatively small size of the college and/or the existence of a pre- 
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screening committee for prior review, which establishes guidelines that the dean and/or 
department head is obliged to follow. 

IMPLICATIONS AND FURTHER RESEARCH 

In other studies of institutional change, regional comprehensive universities have proven 
to be prone to mission drift, or the tendency for institutional focus or identities to become less 
focused or distinct than other institutional types (Henderson, 2009; O’Meara, 2005). Multiple 
scholars have attested to the critical need to align scholarship expectations with institutional 
mission (regardless of type), but that can be difficult if that mission is unclear or subject to 
frequent changes (O’Meara, 2005). Further, the lack of identity can lead to increased faculty 
workloads, resulting in some of the lowest rates of faculty satisfaction of any institutional type, 
despite rising enrollment and faculty employment trends (Henderson, 2007). 

This concern with workloads that result in highly-splayed legs of the traditional three- 
legged stool (teaching, research, and service) also affects new scholarship. In their analysis of a 
landmark survey of chief academic officers, O’Meara and Rice (2005) identified five barriers to 
the adoption of the Boyer model of scholarship. Among those, they ranked “faculty concerns 
about unrealistic expectations that excel in all areas at the same time” as the fourth largest 
obstacle, one that clearly applies to many regional comprehensive universities, including 
Western Carolina University. The assertion, or reassertion, of service or engagement missions 
has helped many regional comprehensives address the issue of drift and provides an impetus for 
providing incorporating a broader definition of scholarship, particularly one that incorporates the 
scholarship of engagement. This was precisely what inspired Western Carolina University to 
pursue the adoption of the Boyer model, but it is the same scholarship of engagement which can 
provoke the most challenges to traditional forms of scholarship, and, in turn, peer review. 

It should be noted that Boyer scholarship and non-traditional scholarship are not 
synonymous. It is possible to do traditional scholarship across the four facets of the Boyer 
model, and it is also possible to do non-traditional scholarship of discovery, or basic research. 
Boyer focused on recognizing different types of scholarly activity, while the non-traditional 
moniker refers to either a non-traditional scholarly product and/or a non-traditional form of 
scholarly communication. One of the primary findings of this study has been the lack of a clear 
definition for the term “non-traditional” scholarship. In most cases, these new scholarly products 
are referred to largely by what they are not, i.e. printed journal articles or books. This negative 
identity leaves the question of exactly what they are open for debate. For example, in the fine 
and performing arts non-printed scholarship, such as performance or creative works, has long 
been the norm. Does that mean that those faculty have, ironically, been practicing non-traditional 
scholarship for a long time or does it mean that they, too, must find new scholarly products to 
address the other areas of scholarship in the Boyer model? In many ways, non-traditional 
scholarship could be just about anything, from a playground to a festival to a report, to an exhibit 
and so on —ad infinitum, but surely there must exist a sufficiently common core that defines 
these products as scholarship, one that can be delineated and operationalized to enhance 
evaluation and reward. 

In O’Meara and Rice’s survey, “unevenness in applying new criteria within and across 
units” (O’Meara & Rice, 2005) ranked as the fifth obstacle to the adoption of the Boyer model of 
scholarship. At Western Carolina University, the Office of the Provost required only that 
departments provide a mechanism for external review of non-traditional products, but they did 
not otherwise specify how that should be done, leaving the operational task to departments. As 
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this study has shown, there was considerable variation in how departments chose to address new 
forms of peer review. The experiences of Western Carolina University suggest that there is a fine 
balance to be struck between flexibility and consistency in implementing processes for the 
review of new forms and types of scholarship. Without an established national or international 
model to use as a guideline, the departments at Western Carolina University turned towards their 
respective disciplines, their own experiences at other institutions, and to their colleagues and 
peers for guidance in creating systems that fit with both departmental and institutional norms. 

While the departments at Western Carolina University enjoyed considerable flexibility in 
the documentation of external peer review, this did not result in 33 different processes. Rather, 
the results of the study show five distinct models for establishing external peer review. The 
difference between these models revolves around the question of who makes the decisions 
regarding external peer review processes, and departments varied from full faculty responsibility 
(model 1, section 2a) to full administrative responsibility (model 5, section 2e). For institutions 
considering not only the Boyer model, but the inclusion of non-traditional scholarship with or 
without Boyer, these models should prove helpful in shaping campus discussions about the 
possibilities for ownership and decision-making regarding external peer review for non- 
traditional scholarship. 

This study focused on the tenure and promotion documents of each respective department 
at Western Carolina University. As such, the findings are indicative of policy, but are indirect 
indicators, at best, of practice or perception. Because the study was limited to these documents, 
the researchers have chosen not to speculate on the reasons behind the disciplinary patterns that 
emerged, but that will be the subject of future study. Those of us who have served on policy 
committees can attest that policy formation is often influenced by factors such as personality, 
context, interest, and time commitment. The documents themselves are works in progress, and 
many of the departments have continued to revise their practices as more faculty vie for tenure 
and promotion under the new definitions of scholarship. In a cursory examination of approved 
documents for 2011, for example, the researchers found that more departments had moved from 
models 1 (section 2a) and 2 (section 2b) to models 3 (section 2c) and 4 (section 2d), suggesting a 
growing consensus toward shared governance and/or more transparent processes. Even with 
some trends towards consensus, the study captured a piece of what can best be described as a 
moving or evolving target. 

Similarly, it will be useful to continue to monitor these processes as they move into 
second and even third generation contexts. Because Western Carolina University adopted these 
standards in 2008, some second generation issues have already appeared. These include, but are 
not limited to, the difficulty of identifying and training peer reviewers, particularly those who 
come from outside of academia. Western Carolina University has created a peer review task 
force tasked with creating a bank of potential peer reviewers in the community, each of whom 
will have undergone non-discipline specific training in the purposes and norms of external peer 
review. Another second generation issue has been the potential for conflict of interest, 
particularly in cases involving payment (such as consultant contracts) and non-disclosure 
agreements, which can inhibit objective peer review. At Western Carolina University, for 
example, some departments have opted to allow scholarship that results from paid service to be 
recognized, while others have not. 

In the end, the fit between traditional peer review, the Boyer model, and non-traditional 
scholarship is still an awkward one. The experiences of Western Carolina University suggest, on 
the other hand, that practical experience in implementing new guidelines can lend new insights 
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into even the most revered of processes. While no department at Western Carolina University 
suggested the possibility of open peer review, which is largely done post-publication by 
publically-identified reviewers and authors, the process does seem to be gaining ground in the 
publishing world and other institutions may wish to consider it. New forms of scholarship are 
straining the limits of peer review, but the same strain is also pushing academics to throw off 
their own blinders, so to speak, and critically examine the basis and rationale for peer review and 
to ask new and interesting questions about evaluating the quality and rigor of scholarly work. 
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